{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Web Mining (爬虫) 规范性的概念：使用代码模拟用户，批量的发送网络请求（request-data），批量的获取数据（response-data）\n",
    " * 课前思考\n",
    "     * http: 当用户在地址输入了网址，发送请求的过程是什么？（思考，类似Flask）http协议/规则（前修课程：http和https图解）\n",
    "     * http的请求方式有哪几种？\n",
    "     * get： 优势：便捷。 缺点：明文，不安全，参数长度有限制\n",
    "     * post：优势：安全，加密，数据长度没有限制\n",
    "     \n",
    "     * 还记得上节课的“状态狗”练习么？那么请思考下面两个问题：\n",
    "     * requests_html到底需要先准备什么？\n",
    "     * 网页的 “HTTP状态狗 = 树.xpath(\"/html/body/main/div/div/a/img\")  # 当成片语/成语记下来, html物件树，有方法xpath抽数据出来”中提到的xpath是啥？\n",
    "     \n",
    "\n",
    "# 20春_Web数据挖掘\n",
    "# 项目2 by 许智超, 廖汉腾\n",
    "\n",
    "# 本周主要内容：requests模块\n",
    "\n",
    "* 来源:[英文官方文档](https://2.python-requests.org/en/master/)\n",
    "* 来源:[中文官方文档](http://cn.python-requests.org/zh_CN/latest/)\n",
    "\n",
    "\n",
    "# 本周学习目标（5、7、8）：  \n",
    "\n",
    "* parse html \n",
    "* xpath 文档\n",
    "* 设置目标网站url，获取页面中标题信息（title、date、link）\n",
    "\n",
    "* 下周学习目标（6）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "/* 本电子讲义使用之CSS */\n",
       "div.code_cell {\n",
       "    background-color: #e5f1fe;\n",
       "}\n",
       "div.cell.selected {\n",
       "    background-color: #effee2;\n",
       "    font-size: 2rem;\n",
       "    line-height: 2.4rem;\n",
       "}\n",
       "div.cell.selected .rendered_html table {\n",
       "    font-size: 2rem !important;\n",
       "    line-height: 2.4rem !important;\n",
       "}\n",
       ".rendered_html pre code {\n",
       "    background-color: #C4E4ff;   \n",
       "    padding: 2px 25px;\n",
       "}\n",
       ".rendered_html pre {\n",
       "    background-color: #99c9ff;\n",
       "}\n",
       "div.code_cell .CodeMirror {\n",
       "    font-size: 2rem !important;\n",
       "    line-height: 2.4rem !important;\n",
       "}\n",
       ".rendered_html img, .rendered_html svg {\n",
       "    max-width: 45%;\n",
       "    height: auto;\n",
       "    float: right;\n",
       "}\n",
       "/* Gradient transparent - color - transparent */\n",
       "hr {\n",
       "    border: 0;\n",
       "    border-bottom: 1px dashed #ccc;\n",
       "}\n",
       ".emoticon{\n",
       "    font-size: 5rem;\n",
       "    line-height: 4.4rem;\n",
       "    text-align: center;\n",
       "    vertical-align: middle;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<style>\n",
    "/* 本电子讲义使用之CSS */\n",
    "div.code_cell {\n",
    "    background-color: #e5f1fe;\n",
    "}\n",
    "div.cell.selected {\n",
    "    background-color: #effee2;\n",
    "    font-size: 2rem;\n",
    "    line-height: 2.4rem;\n",
    "}\n",
    "div.cell.selected .rendered_html table {\n",
    "    font-size: 2rem !important;\n",
    "    line-height: 2.4rem !important;\n",
    "}\n",
    ".rendered_html pre code {\n",
    "    background-color: #C4E4ff;   \n",
    "    padding: 2px 25px;\n",
    "}\n",
    ".rendered_html pre {\n",
    "    background-color: #99c9ff;\n",
    "}\n",
    "div.code_cell .CodeMirror {\n",
    "    font-size: 2rem !important;\n",
    "    line-height: 2.4rem !important;\n",
    "}\n",
    ".rendered_html img, .rendered_html svg {\n",
    "    max-width: 45%;\n",
    "    height: auto;\n",
    "    float: right;\n",
    "}\n",
    "/* Gradient transparent - color - transparent */\n",
    "hr {\n",
    "    border: 0;\n",
    "    border-bottom: 1px dashed #ccc;\n",
    "}\n",
    ".emoticon{\n",
    "    font-size: 5rem;\n",
    "    line-height: 4.4rem;\n",
    "    text-align: center;\n",
    "    vertical-align: middle;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# xpath\n",
    "\n",
    "* [xpath教程](https://www.w3school.com.cn/xpath/index.asp)\n",
    "\n",
    "* xpath是一门在XML文档中查找信息的 语言\n",
    "\n",
    "* 节点\n",
    "    * 元素、属性、文本、命名空间、文档（根）结点\n",
    "* 节点关系\n",
    "    * 父（parent）\n",
    "    * 子（children）\n",
    "    * 同胞（sibling）\n",
    "    * 先辈（ancestor）\n",
    "    * 后代（descendant）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: requests_file in d:\\anaconda\\lib\\site-packages (1.4.3)\n",
      "Requirement already satisfied: requests>=1.0.0 in d:\\anaconda\\lib\\site-packages (from requests_file) (2.22.0)\n",
      "Requirement already satisfied: six in d:\\anaconda\\lib\\site-packages (from requests_file) (1.12.0)\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in d:\\anaconda\\lib\\site-packages (from requests>=1.0.0->requests_file) (2.8)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in d:\\anaconda\\lib\\site-packages (from requests>=1.0.0->requests_file) (2019.6.16)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in d:\\anaconda\\lib\\site-packages (from requests>=1.0.0->requests_file) (1.24.2)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in d:\\anaconda\\lib\\site-packages (from requests>=1.0.0->requests_file) (3.0.4)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install requests_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "html_doc = \"\"\"\n",
    "<html><head><title>The Dormouse's story</title></head>\n",
    "<body>\n",
    "<p class=\"title\"><b>The Dormouse's story</b></p>\n",
    "\n",
    "<p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
    "<a href=\"http://example.com/elsie\" class=\"sister\" id=\"link1\">Elsie</a>,\n",
    "<a href=\"http://example.com/lacie\" class=\"sister\" id=\"link2\">Lacie</a> and\n",
    "<a href=\"http://example.com/tillie\" class=\"sister\" id=\"link3\">Tillie</a>;\n",
    "and they lived at the bottom of a well.</p>\n",
    "\n",
    "<p class=\"story\">...</p>\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# xpath \n",
    "from requests_html import HTMLSession\n",
    "from requests_file import FileAdapter\n",
    "\n",
    "session = HTMLSession()\n",
    "session.mount('file:///Users/xuzhichao/Desktop/2020下学期课程/许智超课程/_Web数据挖掘/_week02_/html_doc.html', FileAdapter())\n",
    "html_obj = session.get(f'file:///Users/xuzhichao/Desktop/2020下学期课程/许智超课程/_Web数据挖掘/_week02_/html_doc.html')\n",
    "\n",
    "selector = html_obj.html.xpath('//p[@class=\"story\"]/a/@href')\n",
    "# 注意：selector的数据类型\n",
    "selector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "print(html_obj.html.xpath('/html/head'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "### 使用模块\n",
    "import pandas as pd\n",
    "from requests_html import HTMLSession\n",
    "\n",
    "payload = {\n",
    "    \"keyword\":\"文学与传媒学院\",\n",
    "    \"p\":\"1\"\n",
    "}\n",
    "\n",
    "session = HTMLSession()\n",
    "r = session.get(\"http://www.nfu.edu.cn/index.php/home/article/search.html\", params=payload)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 取出中大南方搜索文学与传媒学院的标题\n",
    "# r.html.xpath('//*[@class=\"news_title\"]//a/@title')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 练习（找同学回答）\n",
    "# 1、取出中大南方搜索文学与传媒学院的日期\n",
    "# 2、取出中大南方搜索文学与传媒学院的标题链接"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# requests 获取页面信息\n",
    "* 网络页面的数据\n",
    "    * JSON\n",
    "    * HTML\n",
    "-----------------------\n",
    "\n",
    "* 分清信息传递的你我它\n",
    "    * Request 是你 \n",
    "    * Resopnse 是它\n",
    "-----------------------\n",
    "\n",
    "* 可爱的 http://httpbin.org/ ，测试的首选\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requests GET方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Response [200]>\n",
      "<!DOCTYPE html>\n",
      "<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class=\"bg s_ipt_wr\"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class=\"bg s_btn_wr\"><input type=submit id=su value=百度一下 class=\"bg s_btn\"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&amp;tpl=mn&amp;u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href=\"http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === \"\" ? \"?\" : \"&\")+ \"bdorz_come=1\")+ '\" name=\"tj_login\" class=\"lb\">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style=\"display: block;\">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>&copy;2017&nbsp;Baidu&nbsp;<a href=http://www.baidu.com/duty/>使用百度前必读</a>&nbsp; <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a>&nbsp;京ICP证030173号&nbsp; <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>\n",
      "\n",
      "<!DOCTYPE html>\n",
      "<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>ç¾åº¦ä¸ä¸ï¼ä½ å°±ç¥é</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class=\"bg s_ipt_wr\"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class=\"bg s_btn_wr\"><input type=submit id=su value=ç¾åº¦ä¸ä¸ class=\"bg s_btn\"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>æ°é»</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>å°å¾</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>è§é¢</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>è´´å§</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&amp;tpl=mn&amp;u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>ç»å½</a> </noscript> <script>document.write('<a href=\"http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === \"\" ? \"?\" : \"&\")+ \"bdorz_come=1\")+ '\" name=\"tj_login\" class=\"lb\">ç»å½</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style=\"display: block;\">æ´å¤äº§å</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>å",
      "³äºç¾åº¦</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>&copy;2017&nbsp;Baidu&nbsp;<a href=http://www.baidu.com/duty/>ä½¿ç¨ç¾åº¦åå¿",
      "è¯»</a>&nbsp; <a href=http://jianyi.baidu.com/ class=cp-feedback>æè§åé¦</a>&nbsp;äº¬ICPè¯030173å·&nbsp; <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 1.记得安装 第三方 模块 requests\n",
    "# pip install requests\n",
    "\n",
    "# GET请求\n",
    "import requests\n",
    "\n",
    "# url = 'http://httpbin.org/get'\n",
    "\n",
    "url = \"https://baidu.com\"\n",
    "\n",
    "# content 属性 返回的类型 是bytes\n",
    "resopnse = requests.get(url)\n",
    "# decode 转换\n",
    "data = resopnse.content.decode('utf-8')\n",
    "\n",
    "# text 属性 返回的类型是str，猜测编码。优先使用content\n",
    "text = resopnse.text\n",
    "\n",
    "print(resopnse)\n",
    "print(data)\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'args': {'a': '1', 'b': '2'},\n",
       " 'headers': {'Accept': '*/*',\n",
       "  'Accept-Encoding': 'gzip, deflate',\n",
       "  'Host': 'httpbin.org',\n",
       "  'User-Agent': 'python-requests/2.22.0',\n",
       "  'X-Amzn-Trace-Id': 'Root=1-5e7e1b47-9b86d3febcaeda2fa6c0aeb3'},\n",
       " 'origin': '119.125.71.229',\n",
       " 'url': 'http://httpbin.org/get?a=1&b=2'}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 带参数的get请求\n",
    "r2 = requests.get(\"http://httpbin.org/get\",params = {'a':'1','b':'2'}) \n",
    "r2.json()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 让headers成为习惯"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'User-Agent': 'python-requests/2.22.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}\n",
      "{'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}\n"
     ]
    }
   ],
   "source": [
    "# 我们的requests 请求带着重要的'User-Agent'信息，也成为反爬的第一关注点\n",
    "#让headers请求成为习惯习惯\n",
    "print(r2.request.headers)\n",
    "headers = {\n",
    "    'User-Agent':\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15\"\n",
    "}\n",
    "r3 = requests.get(\"http://httpbin.org/get\",headers=headers,params = {'a':'1','b':'2'}) \n",
    "print(r3.request.headers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 带参数的get请求 例子：搜索浏览器搜索python\n",
    "import requests\n",
    "\n",
    "# url = \"https://cn.bing.com/search?p=python\"\n",
    "url = \"https://cn.bing.com/search?\"\n",
    "headers = {\n",
    "    'User-Agent': \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"\n",
    "}\n",
    "\n",
    "response = requests.get(url,headers=headers,params={\"q\":\"python\"})\n",
    "\n",
    "data = response.content.decode()\n",
    "\n",
    "with open(\"bing.html\",'w') as f:\n",
    "    f.write(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 展示：微信公众号的页面参数传递\n",
    "\n",
    "### 练习 ：中大南方网站带参数获取页面（GET方法）/或者其他的网站\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Request POST 方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'args': {},\n",
       " 'data': '',\n",
       " 'files': {},\n",
       " 'form': {'a': '1'},\n",
       " 'headers': {'Accept': '*/*',\n",
       "  'Accept-Encoding': 'gzip, deflate',\n",
       "  'Content-Length': '3',\n",
       "  'Content-Type': 'application/x-www-form-urlencoded',\n",
       "  'Host': 'httpbin.org',\n",
       "  'User-Agent': 'python-requests/2.22.0',\n",
       "  'X-Amzn-Trace-Id': 'Root=1-5e7e1b49-eb7c88b424550c109f0538dc'},\n",
       " 'json': None,\n",
       " 'origin': '119.125.71.229',\n",
       " 'url': 'http://httpbin.org/post'}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# POST请求\n",
    "r3 = requests.post(\"http://httpbin.org/post\", data ={'a':'1'})\n",
    "r3.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## *附加1: 带cookies请求 （get方法）\n",
    "\n",
    "* 请先查看浏览器 Response Headers中的参数有那些？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\\n  \"cookies\": {\\n    \"token\": \"78901234\", \\n    \"userid\": \"123456\"\\n  }\\n}\\n'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cookies = dict(userid = '123456', token='78901234')\n",
    "# 请注意下面的cookies = cookies 我们传入的是字典噢～后面我们还会遇到\n",
    "r_cookies = requests.get(\"http://httpbin.org/cookies\",cookies=cookies)\n",
    "r_cookies.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## *附加2: 带basic-auth请求 （get方法）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'authenticated': True, 'user': 'zhichao'}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "auth = ('zhichao','123456')\n",
    "r_auth = requests.get(\"http://httpbin.org/basic-auth/zhichao/123456\",auth=auth)\n",
    "r_auth.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## *附加3: status-code 还记得你们之前背的401、402、403、404么？\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "curl -I http://httpbin.org/status/404"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  chorme查看 Hearders，里面有哪些内容？\n",
    "\n",
    "* General\n",
    "* Response Headers\n",
    "* Requests Headers\n",
    "* Query String Parameters\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# 1.获取下请求头\n",
      "{'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}\n",
      "--------------------------------------------------\n",
      "# 2.获取响应头\n",
      "{'Bdpagetype': '1', 'Bdqid': '0xd0634a3000118d68', 'Cache-Control': 'private', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip', 'Content-Type': 'text/html;charset=utf-8', 'Date': 'Wed, 25 Mar 2020 13:06:40 GMT', 'Expires': 'Wed, 25 Mar 2020 13:06:26 GMT', 'P3p': 'CP=\" OTI DSP COR IVA OUR IND COM \", CP=\" OTI DSP COR IVA OUR IND COM \"', 'Server': 'BWS/1.1', 'Set-Cookie': 'BAIDUID=38635E4E6C3E5D86C37C1EA24CFD6B63:FG=1; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com, BIDUPSID=38635E4E6C3E5D86C37C1EA24CFD6B63; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com, PSTM=1585141600; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com, BAIDUID=38635E4E6C3E5D86F5BC1F5806881058:FG=1; max-age=31536000; expires=Thu, 25-Mar-21 13:06:40 GMT; domain=.baidu.com; path=/; version=1; comment=bd, BDSVRTM=0; path=/, BD_HOME=1; path=/, H_PS_PSSID=30963_1433_31170_21100_30823_26350; path=/; domain=.baidu.com', 'Strict-Transport-Security': 'max-age=172800', 'Traceid': '1585141600028884634615015927152603401576', 'X-Ua-Compatible': 'IE=Edge,chrome=1', 'Transfer-Encoding': 'chunked'}\n",
      "--------------------------------------------------\n",
      " 3.获取状态码\n",
      "200\n",
      "--------------------------------------------------\n",
      "<RequestsCookieJar[]>\n",
      "--------------------------------------------------\n",
      "<RequestsCookieJar[<Cookie BAIDUID=38635E4E6C3E5D86F5BC1F5806881058:FG=1 for .baidu.com/>, <Cookie BIDUPSID=38635E4E6C3E5D86C37C1EA24CFD6B63 for .baidu.com/>, <Cookie H_PS_PSSID=30963_1433_31170_21100_30823_26350 for .baidu.com/>, <Cookie PSTM=1585141600 for .baidu.com/>, <Cookie BDSVRTM=0 for www.baidu.com/>, <Cookie BD_HOME=1 for www.baidu.com/>]>\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "class RequestSpider(object):\n",
    "    def __init__(self):\n",
    "\n",
    "        url = 'http://baidu.com'\n",
    "        headers = {\n",
    "            'User-Agent': \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"\n",
    "        }\n",
    "        self.response = requests.get(url,headers=headers)\n",
    "\n",
    "    def run(self):\n",
    "        data = self.response.content\n",
    "        \n",
    "        # 1.获取下请求头\n",
    "        request_headers = self.response.request.headers\n",
    "        print(\"# 1.获取下请求头\")\n",
    "        print(request_headers)\n",
    "        print(\"-\"*50)\n",
    "\n",
    "        # 2.获取响应头\n",
    "        rseponse_headers = self.response.headers\n",
    "        print(\"# 2.获取响应头\")\n",
    "        print(rseponse_headers)\n",
    "        print(\"-\"*50)\n",
    "\n",
    "        # 3.获取状态码 ,大家还记得状态码么？找个同学问下。\n",
    "        code = self.response.status_code\n",
    "        print(\" 3.获取状态码\")\n",
    "        print(code)\n",
    "        print(\"-\"*50)\n",
    "        headers = {\n",
    "            'User-Agent': \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"\n",
    "\n",
    "        }\n",
    "\n",
    "        # 4.请求的cookie\n",
    "        request_cookie = self.response.request._cookies\n",
    "        print(request_cookie)\n",
    "        print(\"-\"*50)\n",
    "\n",
    "        # 5.响应的cookie\n",
    "        response_cookie = self.response.cookies\n",
    "        print(response_cookie)\n",
    "        print(\"-\"*50)\n",
    "        \n",
    "RequestSpider().run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 自动转译\n",
    "import requests\n",
    "\n",
    "url = \"https://cn.bing.com/search?\"\n",
    "headers = {\n",
    "    'User-Agent': \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"\n",
    "}\n",
    "\n",
    "response = requests.get(url,headers=headers,data={\"q\":\"python\"})\n",
    "\n",
    "data = response.content.decode()\n",
    "\n",
    "with open('bing.html', 'w') as f:\n",
    "    f.write(data)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "ename": "SSLError",
     "evalue": "HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /user (Caused by SSLError(SSLError(\"bad handshake: SysCallError(10054, 'WSAECONNRESET')\")))",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mSysCallError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\urllib3\\contrib\\pyopenssl.py\u001b[0m in \u001b[0;36mwrap_socket\u001b[1;34m(self, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname)\u001b[0m\n\u001b[0;32m    455\u001b[0m             \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 456\u001b[1;33m                 \u001b[0mcnx\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdo_handshake\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    457\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[0mOpenSSL\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSSL\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mWantReadError\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\OpenSSL\\SSL.py\u001b[0m in \u001b[0;36mdo_handshake\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m   1914\u001b[0m         \u001b[0mresult\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_lib\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSSL_do_handshake\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_ssl\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1915\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_raise_ssl_error\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_ssl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1916\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\OpenSSL\\SSL.py\u001b[0m in \u001b[0;36m_raise_ssl_error\u001b[1;34m(self, ssl, result)\u001b[0m\n\u001b[0;32m   1638\u001b[0m                     \u001b[1;32mif\u001b[0m \u001b[0merrno\u001b[0m \u001b[1;33m!=\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1639\u001b[1;33m                         \u001b[1;32mraise\u001b[0m \u001b[0mSysCallError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0merrno\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0merrorcode\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0merrno\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1640\u001b[0m                 \u001b[1;32mraise\u001b[0m \u001b[0mSysCallError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m-\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m\"Unexpected EOF\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mSysCallError\u001b[0m: (10054, 'WSAECONNRESET')",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mSSLError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\urllib3\\connectionpool.py\u001b[0m in \u001b[0;36murlopen\u001b[1;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\u001b[0m\n\u001b[0;32m    599\u001b[0m                                                   \u001b[0mbody\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mbody\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mheaders\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 600\u001b[1;33m                                                   chunked=chunked)\n\u001b[0m\u001b[0;32m    601\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\urllib3\\connectionpool.py\u001b[0m in \u001b[0;36m_make_request\u001b[1;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[0;32m    342\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 343\u001b[1;33m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_validate_conn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mconn\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    344\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mSocketTimeout\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mBaseSSLError\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\urllib3\\connectionpool.py\u001b[0m in \u001b[0;36m_validate_conn\u001b[1;34m(self, conn)\u001b[0m\n\u001b[0;32m    838\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mconn\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'sock'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m  \u001b[1;31m# AppEngine might not have  `.sock`\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 839\u001b[1;33m             \u001b[0mconn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    840\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\urllib3\\connection.py\u001b[0m in \u001b[0;36mconnect\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    343\u001b[0m             \u001b[0mserver_hostname\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mserver_hostname\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 344\u001b[1;33m             ssl_context=context)\n\u001b[0m\u001b[0;32m    345\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\urllib3\\util\\ssl_.py\u001b[0m in \u001b[0;36mssl_wrap_socket\u001b[1;34m(sock, keyfile, certfile, cert_reqs, ca_certs, server_hostname, ssl_version, ciphers, ssl_context, ca_cert_dir)\u001b[0m\n\u001b[0;32m    346\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mHAS_SNI\u001b[0m \u001b[1;32mand\u001b[0m \u001b[0mserver_hostname\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 347\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mcontext\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwrap_socket\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msock\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mserver_hostname\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mserver_hostname\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    348\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\urllib3\\contrib\\pyopenssl.py\u001b[0m in \u001b[0;36mwrap_socket\u001b[1;34m(self, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname)\u001b[0m\n\u001b[0;32m    461\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[0mOpenSSL\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSSL\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mError\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 462\u001b[1;33m                 \u001b[1;32mraise\u001b[0m \u001b[0mssl\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSSLError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'bad handshake: %r'\u001b[0m \u001b[1;33m%\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    463\u001b[0m             \u001b[1;32mbreak\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mSSLError\u001b[0m: (\"bad handshake: SysCallError(10054, 'WSAECONNRESET')\",)",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mMaxRetryError\u001b[0m                             Traceback (most recent call last)",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\requests\\adapters.py\u001b[0m in \u001b[0;36msend\u001b[1;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[0;32m    448\u001b[0m                     \u001b[0mretries\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmax_retries\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 449\u001b[1;33m                     \u001b[0mtimeout\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    450\u001b[0m                 )\n",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\urllib3\\connectionpool.py\u001b[0m in \u001b[0;36murlopen\u001b[1;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\u001b[0m\n\u001b[0;32m    637\u001b[0m             retries = retries.increment(method, url, error=e, _pool=self,\n\u001b[1;32m--> 638\u001b[1;33m                                         _stacktrace=sys.exc_info()[2])\n\u001b[0m\u001b[0;32m    639\u001b[0m             \u001b[0mretries\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msleep\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\urllib3\\util\\retry.py\u001b[0m in \u001b[0;36mincrement\u001b[1;34m(self, method, url, response, error, _pool, _stacktrace)\u001b[0m\n\u001b[0;32m    398\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mnew_retry\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mis_exhausted\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 399\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mMaxRetryError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m_pool\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0merror\u001b[0m \u001b[1;32mor\u001b[0m \u001b[0mResponseError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcause\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    400\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mMaxRetryError\u001b[0m: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /user (Caused by SSLError(SSLError(\"bad handshake: SysCallError(10054, 'WSAECONNRESET')\")))",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mSSLError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-6-3ab92c31b1dd>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[1;31m# 这个网站不是html 而是json\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     10\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 11\u001b[1;33m \u001b[0mresponse\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mrequests\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mheaders\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mheaders\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     12\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     13\u001b[0m \u001b[1;31m# data = response.content.decode()\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\requests\\api.py\u001b[0m in \u001b[0;36mget\u001b[1;34m(url, params, **kwargs)\u001b[0m\n\u001b[0;32m     73\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     74\u001b[0m     \u001b[0mkwargs\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msetdefault\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'allow_redirects'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 75\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mrequest\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'get'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mparams\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     76\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     77\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\requests\\api.py\u001b[0m in \u001b[0;36mrequest\u001b[1;34m(method, url, **kwargs)\u001b[0m\n\u001b[0;32m     58\u001b[0m     \u001b[1;31m# cases, and look like a memory leak in others.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     59\u001b[0m     \u001b[1;32mwith\u001b[0m \u001b[0msessions\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0msession\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 60\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0msession\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mmethod\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     61\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     62\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\requests\\sessions.py\u001b[0m in \u001b[0;36mrequest\u001b[1;34m(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)\u001b[0m\n\u001b[0;32m    531\u001b[0m         }\n\u001b[0;32m    532\u001b[0m         \u001b[0msend_kwargs\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msettings\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 533\u001b[1;33m         \u001b[0mresp\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mprep\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0msend_kwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    534\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    535\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mresp\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\requests\\sessions.py\u001b[0m in \u001b[0;36msend\u001b[1;34m(self, request, **kwargs)\u001b[0m\n\u001b[0;32m    644\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    645\u001b[0m         \u001b[1;31m# Send the request\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 646\u001b[1;33m         \u001b[0mr\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0madapter\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mrequest\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    647\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    648\u001b[0m         \u001b[1;31m# Total elapsed time of the request (approximately)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\anaconda\\lib\\site-packages\\requests\\adapters.py\u001b[0m in \u001b[0;36msend\u001b[1;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[0;32m    512\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreason\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0m_SSLError\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    513\u001b[0m                 \u001b[1;31m# This branch is for urllib3 v1.22 and later.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 514\u001b[1;33m                 \u001b[1;32mraise\u001b[0m \u001b[0mSSLError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mrequest\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mrequest\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    515\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    516\u001b[0m             \u001b[1;32mraise\u001b[0m \u001b[0mConnectionError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mrequest\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mrequest\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mSSLError\u001b[0m: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /user (Caused by SSLError(SSLError(\"bad handshake: SysCallError(10054, 'WSAECONNRESET')\")))"
     ]
    }
   ],
   "source": [
    "\n",
    "# json 数据传递的用法\n",
    "import requests\n",
    "import json\n",
    "\n",
    "url = 'https://api.github.com/user'\n",
    "headers = {\n",
    "    'User-Agent': \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"\n",
    "}\n",
    "# 这个网站不是html 而是json\n",
    "\n",
    "response = requests.get(url,headers=headers)\n",
    "\n",
    "# data = response.content.decode()\n",
    "# # data 是str\n",
    "# data_dict = json.loads(data)\n",
    "# print(type(data))\n",
    "# print(data)\n",
    "# print(data_dict)\n",
    "\n",
    "# response.json方法\n",
    "data = response.json()\n",
    "print(data)\n",
    "print(type(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 案例基础练习：中大南方网站内搜索“文传学院”的一页标题、链接、日期\n",
    "## 提升：练习获取所有页的标题、链接、日期\n",
    "## 拔高：获取所有页的链接页面的内容"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### 使用模块\n",
    "import pandas as pd\n",
    "from requests_html import HTMLSession\n",
    "\n",
    "payload = {\n",
    "    \"keyword\":\"文学与传媒学院\",\n",
    "    \"p\":\"1\"\n",
    "}\n",
    "\n",
    "session = HTMLSession()\n",
    "r = session.get(\"http://www.nfu.edu.cn/index.php/home/article/search.html\", params=payload)\n",
    "\n",
    "# 从主页挖出 章标题\n",
    "df= pd.DataFrame( {\n",
    "    \"title\": r.html.xpath('//*[@class=\"news_title\"]//a/@title'),\n",
    "    \"date\": r.html.xpath('//*[@class=\"right-more\"]/text()'),\n",
    "    \"link\": r.html.xpath('//*[@class=\"news_title\"]//a/@href')\n",
    "    } )\n",
    "\n",
    "df    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lxml import etree\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "url_nfu_article = \"http://www.nfu.edu.cn/index.php/home/article/search.html\"\n",
    "\n",
    "headers = {\n",
    "     'User-Agent': \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"\n",
    "}\n",
    "data = {\n",
    "    \"keyword\":\"文学与传媒学院\",\n",
    "    \"p\":\"1\"\n",
    "}\n",
    "\n",
    "content_list = []\n",
    "\n",
    "data[\"p\"] = 1\n",
    "\n",
    "\n",
    "response_nfu = requests.get(url_nfu_article, headers=headers, params=data)\n",
    "\n",
    "html_str = response_nfu.content.decode()\n",
    "# print(html_str)\n",
    "html = etree.HTML(html_str)\n",
    "\n",
    "items = {}\n",
    "items[\"title\"] = html.xpath(\"//*[@class=\\\"news_title\\\"]//a/@title\")\n",
    "items[\"date\"] = html.xpath(\"//*[@class=\\\"right-more\\\"]/text()\")\n",
    "items[\"link\"] = html.xpath(\"//*[@class=\\\"news_title\\\"]//a/@href\")\n",
    "content_list.append(items)\n",
    "\n",
    "\n",
    "df = pd.DataFrame(content_list[0])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 用户登录 （下周提前预告）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# cookie 用户的简单登录\n",
    "import requests\n",
    "\n",
    "# 请求数据的url\n",
    "\n",
    "member_url = \"https://www.yaozh.com/member/\"\n",
    "headers = {\n",
    "    'User-Agent': \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36\"\n",
    "}\n",
    "cookies = \"acw_tc=2f624a2a15840160875311695e6d7e27994af827775b34fed9d55cba726de9; PHPSESSID=vaasotsfptlms3gq68oh6mhd82; _ga=GA1.1.1324687667.1584016181; _gid=GA1.2.1663074566.1584178938; _gat=1; Hm_lpvt_65968db3ac154c3089d7f9a4cbb98c94=1584178941; yaozh_logintime=1584178949; yaozh_user=891498%09xzhichao; yaozh_userId=891498; yaozh_jobstatus=kptta67UcJieW6zKnFSe2JyXnoaZcJZpnpyHnKZxanJT1qeSoMZYoNdzb5taqt3Ny8TMxdBZoKifnZ%2BDn5iorJDVop6Yg3HYnmpnm1pjmJ2dc0b8310b0ddC98d22483bB3Ce53b7a3WkZWclGudV6DXn5VtWamhnsZbbKabZ5ieW2iWcGaUnJqamZSUZ5pXoOE%3D034cf6fc2b2e2e0f6305655401292eef; db_w_auth=758331%09xzhichao; yaozh_uidhas=1; yaozh_mylogin=1584178952; acw_tc=2f624a2a15840160875311695e6d7e27994af827775b34fed9d55cba726de9; _ga=GA1.1.1324687667.1584016181; Hm_lvt_65968db3ac154c3089d7f9a4cbb98c94=1584016089%2C1584178939\"\n",
    "\n",
    "# cookies: (optional) **Dict or **CookieJar object to send with the :class:`Request`\n",
    "# cookies 目前是字符串\n",
    "# 需要字典类型\n",
    "\n",
    "# 方法1 正则匹配将cookie转化为字典（或者全部替换）\n",
    "# (.*)=(.*)\n",
    "# \"\\1\":\"\\2\",\n",
    "cook_dict = {\n",
    "    \"acw_tc\":\"2f624a2a15840160875311695e6d7e27994af827775b34fed9d55cba726de9;\",\n",
    "    \"PHPSESSID\":\"vaasotsfptlms3gq68oh6mhd82;\",\n",
    "    \"_ga\":\"GA1.1.1324687667.1584016181;\",\n",
    "    \"_gid\":\"GA1.2.1663074566.1584178938;\",\n",
    "    \"_gat\":\"1;\",\n",
    "    \"Hm_lpvt_65968db3ac154c3089d7f9a4cbb98c94\":\"1584178941;\",\n",
    "    \"yaozh_logintime\":\"1584178949;\",\n",
    "    \"yaozh_user\":\"891498%09xzhichao;\",\n",
    "    \"yaozh_userId\":\"891498;\",\n",
    "    \"yaozh_jobstatus\":\"kptta67UcJieW6zKnFSe2JyXnoaZcJZpnpyHnKZxanJT1qeSoMZYoNdzb5taqt3Ny8TMxdBZoKifnZ%2BDn5iorJDVop6Yg3HYnmpnm1pjmJ2dc0b8310b0ddC98d22483bB3Ce53b7a3WkZWclGudV6DXn5VtWamhnsZbbKabZ5ieW2iWcGaUnJqamZSUZ5pXoOE%3D034cf6fc2b2e2e0f6305655401292eef;\",\n",
    "    \"db_w_auth\":\"758331%09xzhichao;\",\n",
    "    \"yaozh_uidhas\":\"1;\",\n",
    "    \"yaozh_mylogin\":\"1584178952;\",\n",
    "    \"acw_tc\":\"2f624a2a15840160875311695e6d7e27994af827775b34fed9d55cba726de9;\",\n",
    "    \"_ga\":\"GA1.1.1324687667.1584016181;\",\n",
    "    \"Hm_lvt_65968db3ac154c3089d7f9a4cbb98c94\":\"1584016089%2C1584178939\",\n",
    "}\n",
    "\n",
    "# 方法2\n",
    "cook_dict2 = {}\n",
    "cook_list = cookies.split(\";\")\n",
    "for cookie in cook_list:\n",
    "    cook_dict2[cookie.split(\"=\")[0]] =  cookie.split(\"=\")[1]\n",
    "\n",
    "print(cook_dict2)\n",
    "\n",
    "response = requests.get(member_url,headers=headers,cookies=cook_dict2)\n",
    "\n",
    "data = response.content.decode()\n",
    "\n",
    "with open('cookie.html','w') as f:\n",
    "    f.write(data)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "749px",
    "left": "1125.609375px",
    "top": "110px",
    "width": "281.390625px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
